Accelerated lens distortion correction with near-continuous warping optimization

ABSTRACT

A digital image processing technique, such as an image warping operation, is stored in a pre-computed lookup table (LUT) prior to image processing. The LUT represents a pixel-to-pixel mapping of pixel coordinates in a source image to pixel coordinates in a destination image. For vectors containing only inlier pixels, a fast remap table is generated based on the original LUT. Each SIMD vector listed in the fast remap is indexed to the coordinates of one of the source pixels that maps to one of the destination pixels in the vector. Other SIMD vectors that contain at least one outlier pixel are listed in an outlier index. For each vector indexed in the fast remap, linear vector I/O operations are used to load the corresponding source pixels instead of using scatter/gather vector I/O load operations via the LUT. The remaining outlier pixels are processed using scatter/gather I/O operations via the LUT.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to techniques for digital imageprocessing in single-instruction-multiple-data (SIMD) processingenvironments, and more particularly, to techniques for image warping,including accelerated lens distortion correction and image rectificationin stereo vision applications.

BACKGROUND

In the field of digital image processing and stereo vision, imagewarping is a transformation process based on pixel coordinate mapping(e.g., independently of color values). Image warping may, for example,be used to correct distortion of an image caused by a camera lens, andfor image rectification, which is a transformation process used toproject two or more images onto a common image plane.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows several example images pertaining to the field of digitalimage processing and stereo vision, in accordance with an embodiment ofthe present disclosure.

FIG. 2 shows an example arrangement of vectors in a digital image, inaccordance with an embodiment of the present disclosure.

FIG. 3 shows an example source image and an example destination imagethat results from a geometric image transformation operation performedon the source image, in accordance with an embodiment of the presentdisclosure.

FIG. 4 is a graphical representation of inlier and outlier pixels in anexample image rectification operation, in accordance with an embodimentof the present disclosure.

FIG. 5 is a flow diagram of an example methodology for digital imageprocessing, in accordance with an embodiment of the present disclosure.

FIG. 6 is a flow diagram of a portion of the example methodology of FIG.5, in accordance with an embodiment of the present disclosure.

FIG. 7 is a flow diagram of another portion of the example methodologyof FIG. 5, in accordance with an embodiment of the present disclosure.

FIG. 8 is a block diagram representing an example computing device 1000that may be used to perform any of the techniques as variously describedin this disclosure.

DETAILED DESCRIPTION

Techniques are disclosed for efficient processing of digital images,including techniques for accelerated lens distortion correction andimage rectification in digital stereo vision applications. In anembodiment, additional lookup tables (LUTs) are generated based on apre-computed LUT that represents a pixel-to-pixel mapping of sourcepixel coordinates of an input image to destination pixel coordinates ofan output image. A first of the generated LUTs, also referred to as afast remap table, represents a vector-to-pixel mapping of a firstplurality of vectors of the destination pixel coordinates to a firstplurality of the source pixel coordinates based on the pre-computed LUT,where all of the pixel-to-pixel mappings in the pre-computed LUT arelinear with a slope of one (1) for all of the destination pixelcoordinates in each of the first plurality of vectors. The secondgenerated LUT, also referred to as an outlier index, represents alisting of a second plurality of vectors of the destination pixelcoordinates, where at least one of the pixel-to-pixel mappings in thepre-computed LUT is not linear with slope 1 for any of the destinationpixel coordinates in each of the second plurality of vectors. A linearremapping operation from a source image data is then applied to each ofthe destination pixel coordinates in the first plurality of vectors inthe fast remap table to obtain a first portion of destination imagedata, and a non-linear remapping operation from the source image data isapplied to each of the destination pixel coordinates in the secondplurality of vectors in the outlier index to obtain a second portion ofthe destination image data.

General Overview

Digital image warping functions, including lens distortion correctionand image rectification, are computationally complex to solve. Forinstance, in some digital image processing techniques, the value of anoutput pixel is computed as a function of one or more input pixels. Apre-computed lookup table (LUT) can be used to map coordinates of pixelsin a source image to coordinates of pixels in a destination image. Toevaluate each pixel individually, first a transformation value isretrieved from the LUT to obtain the coordinates of the source pixelthat corresponds to the coordinates of the destination pixel. Dependingon the warping function, the source pixel coordinates may not belinearly related for a given vector of destination pixel coordinates. Asused in this disclosure, the term “vector” refers to a one-dimensionaldata array. Therefore, a scatter/gatherer input/output (I/O) techniqueis employed to load pixel data in the source image on a pixel-by-pixelbasis for vector processing. While use of a pre-computed LUT can reducethe computational load at run time, scatter/gather I/O is less efficientthan linear I/O operations since scatter/gather uses an operationallyintensive per-pixel addressing scheme.

To this end, and in accordance with an embodiment of the presentdisclosure, it is appreciated that while the overall image warpingfunction may not be linear in many cases, at a local scale (e.g., thelength of a SIMD vector), the pixel-to-pixel coordinate mapping isnearly linear (e.g., the slope is 1) for so-called inlier pixels. Totake advantage of this linearity, an image transformation operation,such as an image warping function, can be represented by a pre-computedlookup table (LUT) prior to image processing. The pre-computed LUTrepresents a pixel-to-pixel mapping between pixel coordinates in asource (input) image and pixel coordinates in a destination (result)image. For a given SIMD vector of horizontally adjacent pixels in thedestination image (e.g., pixels in several consecutive columns of thesame row), at least some of the pixel coordinates in the source imagethat map to the destination pixels in the vector are also horizontallyadjacent. These are referred to as inlier pixels because thetransformations of pixels within the same vector are linear with slope1, or near linear for non-integer mappings. By contrast, an outlierpixel is any pixel for which the pixel-to-pixel mapping in thepre-computed LUT is non-linear with respect to the mappings of otherpixels in the same vector. For vectors containing only inlier pixels, asecond, fast remap table is generated based on the original,pre-computed LUT. The fast remap table represents SIMD vectorscontaining only inlier pixels. For example, for a given SIMD vector of16 bytes (e.g., four RGBA pixels of data, where each RGBA pixel consumesfour bytes of which three are color channels and one is an unused alphachannel), four horizontally adjacent pixels in the destination image mapfrom four horizontally adjacent pixels in the source image. Each SIMDvector listed in the fast remap table is indexed to the coordinates ofone of the source pixels that maps to one of the destination pixels inthe vector. Other SIMD vectors containing at least one outlier pixel arelisted in an outlier index. For each vector indexed in the fast remaptable, linear vector I/O operations are used at run time to load thecorresponding source pixels instead of using the relatively expensivescatter/gather vector I/O load operations via the pre-computed LUT. Theremaining outlier pixels are processed using scatter/gather vector I/Ooperations via the original, pre-computed LUT. Numerous variations andconfigurations will be apparent in light of this disclosure.

FIG. 1 shows several example images pertaining to the field of digitalimage processing and stereo vision, in accordance with an embodiment ofthe present disclosure. Source images 100 a and 100 b are two imagestaken from two different perspectives. An image warping function, suchas lens distortion correction and image rectification, can be used toremove distortions caused by the camera lens, which cause straight linesto appear rounded, and to perform an affine transform that aligns twostereo images so that the horizontal lines in one of the imagescorrespond to the horizontal lines in the other image. Destinationimages 102 a and 102 b are the results of such an image warping functionas applied to source images 100 a and 100 b, respectively. Note thealignment of the horizontal lines in images 102 a and 102 b.

If an image transformation operation is non-linear across many or allpixels {x, y}, then a function F ({x, y}) that defines the imagetransformation may be difficult to compute. Since a warping function,such as described with respect to FIG. 1, can be highly non-linear(e.g., the lens distortion may involve a polynom of height ˜7 or more),one technique for performing the warping function includes creating aLUT or remap table that is computed once, and then applying a so-calledgeneric remapping operation for every input image using the pre-computedLUT. The LUT represents a pixel-to-pixel mapping of pixel coordinates ina source image to pixel coordinates in a destination image. An exampleof an existing warping function includes a geometric imagetransformation, such as OpenCV initUndistortRectifyMap( ) and remap( ).The OpenCV initUndistortRectifyMap( ) function builds a mapping ofdestination pixel coordinates to source pixel coordinates for theinverse mapping algorithm that is subsequently used by the remap( )function. In other words, for each pixel in the corrected and rectifieddestination image, the initUndistortRectifyMap( ) function computes thecorresponding pixel coordinates in the original source image from acamera.

In terms of performance, such generic remapping operations arerelatively slow as they involve two input/output (I/O) steps. Forexample, an intermediate per-pixel read from the pre-computed LUT isperformed. Next, the per-pixel coordinates obtained from the LUT areused to load pixel data in the source image using scatter/gather vectorI/O. Next, a bilinear interpolation is applied to each pixel coordinatebased on the source pixel data, and the interpolated source pixels arewritten to the destination coordinates specified by the LUT. However, inaccordance with an embodiment, it is appreciated that, at a local scale(e.g., a SIMD vector length), the function F({x, y}) that defines theimage transformation at the local scale for a subset of {x, y} is linearor nearly linear (e.g., the slope of the transformations is 1).

An image processing function that implements a point operation on animage (e.g., an image warping operation performed on several pixels thatdepends on only one of those pixels) can be mapped to a SIMD processorand efficiently chained. This is because the order of the pixelspresented to each SIMD processor is unimportant as each of the resultingdestination pixels in the vector only depend on one source pixel. Whenperforming an image warping operation on those portions of an imagewhere the integral part of the image warping function is linear withslope 1, linear vector I/O operations (point operations) can replacescatter/gather vector I/O operations, and thus reduce the total numberof vector I/O operations.

FIG. 2 shows an example image 200 containing a plurality of pixels 202(not drawn to scale). For example, the image may include 1024 columns ofpixels and 768 rows of pixels, for a total of 786,432 pixels. Inaccordance with an embodiment of the present disclosure, for vectorprocessing of the image data, each pixel 202 is grouped into a vector204 of a specified length. For example, as shown in FIG. 1, each vector204 includes four pixels (p0, p1, p2, p3) of data.

FIG. 3 shows an example source image 310 and an example destinationimage 320 (not drawn to scale) that results from a geometric imagetransformation operation performed on the source image 310, inaccordance with an embodiment of the disclosure. In FIG. 3, horizontalrows of pixels in the source image 310 and the destination image 320 aredenoted along the x axis, and vertical columns of pixels are denotedalong the y axis, with the upper-leftmost pixel in each image havingcoordinates (x=0, y=0). The image transformation operation in thisexample is a pixel-to-pixel mapping of pixel coordinates in the sourceimage 310 to pixel coordinates in the destination image 320. Themappings are represented by arrows from pixels in the source image 310to pixels in the destination image 320 (e.g., the arrow generallyindicated at 330). The mappings are further described in the tables inFIG. 3 for two vectors of pixels in the destination image 320 (Vector 1and Vector 2). For example, the coordinates of the pixels in Vector 1 ofthe destination image 320 map from the coordinates of some of the pixelsin the source image 310 as follows:

TABLE 1 Linear Pixel Mappings, Vector 1 Source Destination (0, 0) (5, 3)(1, 0) (6, 3) (2, 0) (7, 3) (3, 0) (8, 3)

The mappings of the four pixels in Table 1 are linear and comply withthe following equation:

F({x+i, y})=F({x, y})+{i,0}  (1)

In Equation (1), F({x, y}) is a function that represents the integralpart of mappings for each pixel in the destination image 320, and i isan offset variable. For a vector of pixels [(x_(s), y_(s)), (x_(s)+1,y_(s)), . . . (x_(s)+n,y_(s))], where, x_(s) and y_(s) are pixelcoordinates in the source image and n is the length of a vector, eachpixel in the vector that complies with Equation (1) is referred to as aninlier pixel.

When performing an image warping operation on vectors that contain onlyinlier pixels that comply with Equation (1), such as the example Vector1 in Table 1, the vectors of pixels in the destination image 220 thatcontain only inlier pixels are indexed in a fast remap table. The fastremap table relates the respective vector to the coordinates of thefirst source pixel in the respective mapping 330 between the sourceimage 210 and the destination image 220. For example, referring to Table1, the fast remap table indexes Vector 1 to source pixel coordinates(0,0) because all of the destination pixels in Vector 1 are inlierpixels. Subsequently, each destination pixel of a vector indexed in thefast remap table is processed using a linear vector I/O load of sourcepixel coordinates indexed from the first source pixel coordinate in thefast remap table (e.g., for a vector of length 4, pixel data for thefirst pixel (0,0) and the three adjacent pixels (1,0), (2,0) and (3,0)are loaded without referencing the pixel-to-pixel mapping in thepre-computed LUT). This is in contrast to a scatter/gather vector I/Oload of source pixel data, in which the pre-computed LUT is referencedto obtain the corresponding source pixel coordinates for eachdestination pixel coordinate in the vector, and then the respectivesource pixel data is loaded.

Referring still to FIG. 3, the coordinates of the pixels in Vector 2 ofthe destination image 220 map from the coordinates of some of the pixelsin the source image 210 as follows:

TABLE 2 Non-linear Pixel Mappings, Vector 2 Source Destination (0, 8)(1, 15) (1, 8) (2, 15) (2, 8) (3, 15) (3, 8) (4, 14)

In contrast to the linear mappings of Table 1, the mappings in Table 2are non-linear and do not comply with Equation (1) because the mappingof the fourth destination pixel coordinates (4,14) in Vector 2 isnon-linear with respect to the mappings of the first three destinationpixel coordinates in the same vector: (1,15), (2,15) and (3,15). Assuch, the fourth destination pixel in Vector 2 is an outlier pixel.Thus, because Vector 2 includes at least one outlier pixel, linear I/Ooperations cannot be used with any pixel in Vector 2. Instead, Vector 2is added to an index of outlier vectors. All vectors in the index ofoutlier pixels (e.g., Vector 2) are subsequently processed using, forexample, a generic remap operation, such as OpenCVinitUndistortRectifyMap( ) and remap( ) rather than using a linearvector I/O load described above with respect to the inlier vectors.

FIG. 4 is a graphical representation 400 of inlier and outlier pixels inan example VGA image rectification operation used for images produced bya DS4 stereo camera device, in accordance with an embodiment of thepresent disclosure. In the graphical representation 400, black pixelsrepresent inlier pixels, and white pixels represent outlier pixels. Ascan be seen, the outlier pixels are scarce (e.g., 5% or less of allpixels), and therefore the overall performance impact of processing theoutlier pixels using only the generic remap operation is marginal.

In either case, whether or not the vector contains outlier pixels, theimage warping function includes resampling of the source pixels byinterpolation. Interpolation is a mathematical method of reproducingmissing values from a set of known values. Some image interpolationtechniques include the nearest neighbor, bilinear interpolation,quadratic interpolation, cubic interpolation and spline interpolation.Bilinear interpolation can be used to determine a missing value bytaking a weighted average of the surrounding known values.

Example Methodology

FIG. 5 is a flow diagram of an example methodology 500 for imageprocessing, in accordance with an embodiment of the present disclosure.All or portions of the methodology 500 may be implemented, for example,by a computing device 1000, such as described with respect to FIG. 8.The methodology 500 may be split into two phases: initialization 502 andrun time 504. FIG. 6 is a flow diagram of the initialization phaseportion 502 of the example methodology 500 of FIG. 5, in accordance withan embodiment of the present disclosure. FIG. 7 is a flow diagram of therun-time phase portion 504 of the example methodology 500 of FIG. 5, inaccordance with an embodiment of the present disclosure.

During the initialization phase 502, as shown in FIGS. 5 and 6, apre-computed image transformation LUT 510 is received 520. Thepre-computed LUT 510 represents a pixel-to-pixel mapping of pixelcoordinates in a source image to pixel coordinates in a destinationimage. Such a mapping may represent, for example, an image warpingoperation (e.g., lens distortion correction and stereo vision imagerectification). Based on the pre-computed LUT 510, a fast remap table512 is generated 522 for vectors with inlier pixels. The fast remaptable 512 is generated based on the size of an SIMD vector that will beused for processing the input image. The pixel coordinates are groupedinto vectors of the specified size. Each vector is assigned a vectornumber. For vectors in which the pixel-to-pixel mappings comply with theinlier pixel linear locality condition defined by Equation (1), thecorresponding vector number and the coordinates of the first sourcepixel are stored in the fast remap table 512. In addition to the fastremap table 512, an outlier index 514 is generated 524 based on the sizeof the SIMD vector. For vectors in which at least one of thepixel-to-pixel mappings does not comply with Equation (1), thecorresponding vector number is stored in the outlier index 514. Theprocess of generating the fast remap table 512 and the outlier index 514is performed for all vectors.

During the run time phase 504, as shown in FIGS. 4 and 6 a source image516 is received 530, and a fast remap operation is performed 532 on thesource image 516 to obtain and output 534 a destination image 518 havingthe image transformation defined by the pre-computed LUT 510. The fastremap operation 532 includes processing the vectors with inlier pixelsusing the fast remap table 512, and processing the vectors in theoutlier index using the pre-computed LUT 510. The fast remap operationis performed for all vectors. The run time phase 504 operations can berepeated any number of times for any number of source images 516.

The following table lists experimental performance results comparing anexisting image transformation operation to a new image transformationoperation according to an embodiment of the present disclosure. Thefunctions compared are rectification of different image sizes. Allimages are in RGBA format (each pixel consumes four bytes: three colorchannels and one unused alpha channel). With respect to this table, theterm “rectification” refers to the following two image warpingoperations combined: lens distortion correction, and affine transform,ensuring lines in each image correspond to each other (e.g., such asshown in FIG. 1). The performance factor represents the approximatefactor of improvement in speed for each experimental rectification.

TABLE 3 Experimental Results, RGBA Image Rectification New Rectify SizeExisting (milli- Performance (Input → Output) (milliseconds) seconds)factor 1920 × 1080 → 640 × 360 6.64 1.28 5.17 1920 × 1080 → 1280 × 72023.70 5.28 4.49 1920 × 1080 → 1920 × 1080 53.76 5.96 9.03 640 × 480 →320 × 240 2.74 0.34 8.12 640 × 480 → 640 × 480 9.12 0.84 10.79

Example System

FIG. 8 is a block diagram representing an example computing device 1000that may be used to perform any of the techniques as variously describedin this disclosure. For example, the methodologies of FIGS. 5, 6, 7, 8,or any portions thereof, may be implemented in the computing device1000. The computing device 1000 may be any computer system, such as aworkstation, desktop computer, server, laptop, handheld computer, tabletcomputer (e.g., the iPad™ tablet computer), mobile computing orcommunication device (e.g., the iPhone™ mobile communication device, theAndroid™ mobile communication device, and the like), or other form ofcomputing or telecommunications device that is capable of communicationand that has sufficient processor power and memory capacity to performthe operations described in this disclosure. A distributed computationalsystem may be provided comprising a plurality of such computing devices.

The computing device 1000 includes one or more storage devices 1010and/or non-transitory computer-readable media 1020 having encodedthereon one or more computer-executable instructions or software forimplementing techniques as variously described in this disclosure. Thestorage devices 1010 may include a computer system memory or randomaccess memory, such as a durable disk storage (which may include anysuitable optical or magnetic durable storage device, e.g., RAM, ROM,Flash, USB drive, or other semiconductor-based storage medium), ahard-drive, CD-ROM, or other computer readable media, for storing dataand computer-readable instructions and/or software that implementvarious embodiments as taught in this disclosure. The storage device1010 may include other types of memory as well, or combinations thereof.The storage device 1010 may be provided on the computing device 1000 orprovided separately or remotely from the computing device 1000. Thenon-transitory computer-readable media 1020 may include, but are notlimited to, one or more types of hardware memory, non-transitorytangible media (for example, one or more magnetic storage disks, one ormore optical disks, one or more USB flash drives), and the like. Thenon-transitory computer-readable media 1020 included in the computingdevice 1000 may store computer-readable and computer-executableinstructions or software for implementing various embodiments. Thecomputer-readable media 1020 may be provided on the computing device1000 or provided separately or remotely from the computing device 1000.

The computing device 1000 also includes at least one processor 1030 forexecuting computer-readable and computer-executable instructions orsoftware stored in the storage device 1010 and/or non-transitorycomputer-readable media 1020 and other programs for controlling systemhardware. Virtualization may be employed in the computing device 1000 sothat infrastructure and resources in the computing device 1000 may beshared dynamically. For example, a virtual machine may be provided tohandle a process running on multiple processors so that the processappears to be using only one computing resource rather than multiplecomputing resources. Multiple virtual machines may also be used with oneprocessor.

A user may interact with the computing device 1000 through an outputdevice 1040, such as a screen or monitor, which may display imagesprovided in accordance with some embodiments. The output device 1040 mayalso display other aspects, elements and/or information or dataassociated with some embodiments. The computing device 1000 may includeother I/O devices 1050 for receiving input from a user, for example, akeyboard, a joystick, a game controller, a pointing device (e.g., amouse, a user's finger interfacing directly with a display device,etc.), or any suitable user interface. The computing device 1000 mayinclude other suitable conventional I/O peripherals, such as a camera1052. The computing device 1000 can include and/or be operativelycoupled to various suitable devices for performing one or more of thefunctions as variously described in this disclosure.

The computing device 1000 may run any operating system, such as any ofthe versions of Microsoft® Windows® operating systems, the differentreleases of the Unix and Linux operating systems, any version of theMacOS® for Macintosh computers, any embedded operating system, anyreal-time operating system, any open source operating system, anyproprietary operating system, any operating systems for mobile computingdevices, or any other operating system capable of running on thecomputing device 1000 and performing the operations described in thisdisclosure. In an embodiment, the operating system may be run on one ormore cloud machine instances.

Various embodiments can be implemented using hardware elements, softwareelements, or a combination of both. Examples of hardware elementsincludes processors, microprocessors, circuits, circuit elements (e.g.,transistors, resistors, capacitors, inductors, and so forth), integratedcircuits, application specific integrated circuits (ASIC), programmablelogic devices (PLD), digital signal processors (DSP), field programmablegate array (FPGA), logic gates, registers, semiconductor device, chips,microchips, chip sets, and so forth. Examples of software may includesoftware components, programs, applications, computer programs,application programs, system programs, machine programs, operatingsystem software, middleware, firmware, software modules, routines,subroutines, functions, methods, procedures, software interfaces,application program interfaces (API), instruction sets, computing code,computer code, code segments, computer code segments, words, values,symbols, or any combination thereof. Whether hardware elements and/orsoftware elements are used may vary from one embodiment to the next inaccordance with any number of factors, such as desired computationalrate, power levels, heat tolerances, processing cycle budget, input datarates, output data rates, memory resources, data bus speeds and otherdesign or performance constraints.

Some embodiments may be implemented, for example, using amachine-readable medium or article which may store an instruction or aset of instructions that, if executed by a machine, may cause themachine to perform a method and/or operations in accordance with anembodiment of the present disclosure. Such a machine may include, forexample, any suitable processing platform, computing platform, computingdevice, processing device, computing system, processing system,computer, processor, or the like, and may be implemented using anysuitable combination of hardware and software. The machine-readablemedium or article may include, for example, any suitable type of memoryunit, memory device, memory article, memory medium, storage device,storage article, storage medium and/or storage unit, for example,memory, removable or non-removable media, erasable or non-erasablemedia, writeable or re-writeable media, digital or analog media, harddisk, floppy disk, Compact Disk Read Only Memory (CD-ROM), Compact DiskRecordable (CD-R), Compact Disk Rewriteable (CD-RW), optical disk,magnetic media, magneto-optical media, removable memory cards or disks,various types of Digital Versatile Disk (DVD), a tape, a cassette, orthe like. The instructions may include any suitable type of executablecode implemented using any suitable high-level, low-level,object-oriented, visual, compiled and/or interpreted programminglanguage.

Unless specifically stated otherwise, it will be appreciated that termssuch as “processing,” “computing,” “calculating,” “determining,” or thelike, refer to the action and/or processes of a computer or computingsystem, or similar electronic computing device, that manipulates and/ortransforms data represented as physical quantities (e.g., electronic)within the computing system's registers and/or memories into other datasimilarly represented as physical quantities within the computingsystem's memories, registers or other such information storage,transmission or displays.

As will be appreciated in light of this disclosure, the various modulesand components of the system can be implemented in software, such as aset of instructions (e.g., HTML, XML, C, C++, object-oriented C,JavaScript, Java, BASIC, etc.) encoded on any computer readable mediumor computer program product (e.g., hard drive, server, disc, or othersuitable non-transient memory or set of memories), that when executed byone or more processors, cause the various methodologies provided in thisdisclosure to be carried out. It will be appreciated that, in someembodiments, various functions performed by the user computing system,as described in this disclosure, can be performed by similar processorsand/or databases in different configurations and arrangements, and thatthe depicted embodiments are not intended to be limiting. Variouscomponents of this example embodiment, including the computing device1000, can be integrated into, for example, one or more desktop or laptopcomputers, workstations, tablets, smart phones, game consoles, set-topboxes, or other such computing devices. Other componentry and modulestypical of a computing system, such as processors (e.g., centralprocessing unit and co-processor, graphics processor, etc.), inputdevices (e.g., keyboard, mouse, touch pad, touch screen, etc.), andoperating system, are not shown but will be readily apparent.

Further Example Embodiments

The following examples pertain to further embodiments, from whichnumerous permutations and configurations will be apparent.

Example 1 is a computer-implemented image processing method, comprising:receiving, by a computer processor, a first lookup table representing apixel-to-pixel mapping of source pixel coordinates of an input image todestination pixel coordinates of an output image; generating, by thecomputer processor, a second lookup table representing a vector-to-pixelmapping of a first plurality of vectors of the destination pixelcoordinates to a first plurality of the source pixel coordinates basedon the first lookup table, where all of the destination pixelcoordinates in each of the first plurality of vectors correspond topixel-to-pixel mappings in the first lookup table that are linear withslope 1 with respect to each other; and generating, by the computerprocessor, a third lookup table representing a listing of a secondplurality of vectors of the destination pixel coordinates, where atleast one of the destination pixel coordinates in each of the secondplurality of vectors correspond to a pixel-to-pixel mapping in the firstlookup table that is not linear with slope 1 with respect to otherpixel-to-pixel mappings.

Example 2 includes the subject matter of claim 1, where the methodcomprises receiving, by the computer processor, source image data;applying, by the computer processor, a linear remapping operation fromthe source image data to each of the destination pixel coordinates inthe first plurality of vectors in the second lookup table to obtain afirst portion of destination image data; and applying, by the computerprocessor, a non-linear remapping operation from the source image datato each of the destination pixel coordinates in the second plurality ofvectors in the third lookup table to obtain a second portion of thedestination image data.

Example 3 includes the subject matter of Example 2, where the linearremapping operation comprises retrieving, by the computer processor andfrom the source image data, first source pixel data corresponding to arespective one of the first plurality of vectors in the second lookuptable based on the vector-to-pixel mapping in the second lookup table.

Example 4 includes the subject matter of Example 3, where the methodcomprises performing, by the computer processor, an interpolation on thefirst source pixel data to obtain the first portion of the destinationimage data.

Example 5 includes the subject matter of any of Examples 2-4, where thenon-linear transformation operation comprises retrieving, by thecomputer processor and from the source image data, second source pixeldata corresponding to a respective one of the second plurality ofvectors in the third lookup table based on the pixel-to-pixel mapping inthe first lookup table.

Example 6 includes the subject matter of Example 4, where the methodcomprises performing, by the computer processor, an interpolation on thesecond source pixel data to obtain the second portion of the destinationimage data.

Example 7 includes the subject matter of any of Examples 1-6, where allof the pixel-to-pixel mappings in the first lookup table comply with thefollowing equation for all of the destination pixel coordinates in eachof the first plurality of vectors: F({x+i,y})=F({x,y})+{i,0}.

Example 8 includes the subject matter of any of Examples 1-7, where eachof the first and second pluralities of vectors comprises at least twoadjacent destination pixel coordinates.

Example 9 is a non-transient computer program product havinginstructions encoded thereon that when executed by one or moreprocessors cause a process to be carried out, the process comprising:receiving a first lookup table representing a pixel-to-pixel mapping ofsource pixel coordinates of an input image to destination pixelcoordinates of an output image; generating a second lookup tablerepresenting a vector-to-pixel mapping of a first plurality of vectorsof the destination pixel coordinates to a first plurality of the sourcepixel coordinates based on the first lookup table, where all of thepixel-to-pixel mappings in the first lookup table are linear with slope1 for all of the destination pixel coordinates in each of the firstplurality of vectors; generating a third lookup table representing alisting of a second plurality of vectors of the destination pixelcoordinates, where any of the pixel-to-pixel mappings in the firstlookup table are not linear with slope 1 for any of the destinationpixel coordinates in each of the second plurality of vectors; receivingsource image data; applying a linear remapping operation from the sourceimage data to each of the destination pixel coordinates in the firstplurality of vectors in the second lookup table to obtain a firstportion of destination image data; and applying a non-linear remappingoperation from the source image data to each of the destination pixelcoordinates in the second plurality of vectors in the third lookup tableto obtain a second portion of the destination image data.

Example 10 includes the subject matter of Example 9, where the linearremapping operation comprises retrieving, from the source image data,first source pixel data corresponding to a respective one of the firstplurality of vectors in the second lookup table based on thevector-to-pixel mapping in the second lookup table.

Example 11 includes the subject matter of Example 10, where the processcomprises performing an interpolation on the first source pixel data toobtain the first portion of the destination image data.

Example 12 includes the subject matter of any of Examples 9-11, wherethe non-linear transformation operation comprises retrieving, from thesource image data, second source pixel data corresponding to arespective one of the second plurality of vectors in the third lookuptable based on the pixel-to-pixel mapping in the first lookup table.

Example 13 includes the subject matter of Example 12, where the processcomprises performing, by the computer processor, an interpolation on thesecond source pixel data to obtain the second portion of the destinationimage data.

Example 14 includes the subject matter of any of Examples 9-13, whereall of the pixel-to-pixel mappings in the first lookup table comply withthe following equation for all of the destination pixel coordinates ineach of the first plurality of vectors: F({x+i,y})=F({x,y})+{i,0}.

Example 15 includes the subject matter of any of Examples 9-14, whereeach of the first and second pluralities of vectors comprises at leasttwo adjacent destination pixel coordinates.

Example 16 is a system comprising: a storage; one or more computerprocessors operatively coupled to the storage, the one or more computerprocessors configured to execute instructions stored in the storage thatwhen executed cause any of the one or more computer processors to carryout a process comprising: receiving a first lookup table representing apixel-to-pixel mapping of source pixel coordinates of an input image todestination pixel coordinates of an output image; generating a secondlookup table representing a vector-to-pixel mapping of a first pluralityof vectors of the destination pixel coordinates to a first plurality ofthe source pixel coordinates based on the first lookup table, where allof the pixel-to-pixel mappings in the first lookup table are linear withslope 1 for all of the destination pixel coordinates in each of thefirst plurality of vectors; generating a third lookup table representinga listing of a second plurality of vectors of the destination pixelcoordinates, where any of the pixel-to-pixel mappings in the firstlookup table are not linear with slope 1 for any of the destinationpixel coordinates in each of the second plurality of vectors; receivingsource image data; applying a linear remapping operation from the sourceimage data to each of the destination pixel coordinates in the firstplurality of vectors in the second lookup table to obtain a firstportion of destination image data; and applying a non-linear remappingoperation from the source image data to each of the destination pixelcoordinates in the second plurality of vectors in the third lookup tableto obtain a second portion of the destination image data.

Example 17 includes the subject matter of Example 16, where the linearremapping operation comprises retrieving, from the source image data,first source pixel data corresponding to a respective one of the firstplurality of vectors in the second lookup table based on thevector-to-pixel mapping in the second lookup table.

Example 18 includes the subject matter of Example 17, where the processcomprises performing an interpolation on the first source pixel data toobtain the first portion of the destination image data.

Example 19 includes the subject matter of any of Example 16-18, wherethe non-linear transformation operation comprises retrieving, from thesource image data, second source pixel data corresponding to arespective one of the second plurality of vectors in the third lookuptable based on the pixel-to-pixel mapping in the first lookup table.

Example 20 includes the subject matter of Example 19, where the processcomprises performing, by the computer processor, an interpolation on thesecond source pixel data to obtain the second portion of the destinationimage data.

Example 21 includes the subject matter of any of Example 16-20, whereall of the pixel-to-pixel mappings in the first lookup table comply withthe following equation for all of the destination pixel coordinates ineach of the first plurality of vectors: F({x+i,y})=F({x,y})+{i,0}.

Example 22 is a computer-implemented image processing method,comprising: receiving, by a computer processor, a first lookup tableincluding a plurality of transform values, each of the transform valuesrepresenting a pixel-to-pixel mapping of destination pixel coordinatesof an output image from source pixel coordinates of an input image inwhich a geometrical distortion of the input image is modified by themapping; for each of a plurality of vectors in the first lookup table,calculating, by the computer processor, a slope between the transformvalue corresponding to a first one of the destination pixel coordinatesin the respective vector and the transform value corresponding to asecond one of the destination pixel coordinates in the same vector;generating, by the computer processor, a second lookup table including aplurality of inlier pixel indices, each of the inlier pixel indicesrepresenting an association between: (i) one of the vectors for whichthe calculated slope is equal to 1, and (ii) the source pixelcoordinates associated, via the pixel-to-pixel mapping, with the firstone of the destination pixel coordinates in the respective vector;generating, by the computer processor, a third lookup table including aplurality of outlier pixel coordinates, each of the outlier pixelcoordinates representing one of the destination pixel coordinates in oneof the vectors for which the calculated slope is not equal to 1; foreach of the vectors in the second lookup table: loading, by the computerprocessor, the source pixel coordinates associated, via the inlier pixelindices, with the respective vector into asingle-instruction-multiple-data (SIMD) vector; and applying, by thecomputer processor, a bilinear interpolation to the SIMD vector toobtain the corresponding one of the destination pixel coordinates; andfor each of the outlier pixel coordinates in the third lookup table:remapping, by the computer processor, the respective source pixelcoordinates using the transform value in the first lookup tablecorresponding to the respective outlier pixel coordinates to obtain thecorresponding one of the destination pixel coordinates.

Example 23 includes the subject matter of Example 22, where each of thepixel-to-pixel mappings associated with the vectors for which thecalculated slope is equal to 1 comply with the following equation:F({x+i,y})=F({x,y})+{i,0}.

Example 24 includes the subject matter of any of Examples 22-23, whereeach of the vectors comprises at least two adjacent destination pixelcoordinates.

The foregoing description of example embodiments has been presented forthe purposes of illustration and description. This description is notintended to be exhaustive or to limit the present disclosure to theprecise forms disclosed. Many modifications and variations are possiblein light of this disclosure. This disclosure does not intend to limitthe scope of the various embodiments. Future filed applications claimingpriority to this application may claim the disclosed subject matter in adifferent manner, and may generally include any set of one or morelimitations as variously disclosed or otherwise demonstrated herein.

What is claimed is:
 1. A computer-implemented image processing method,comprising: receiving, by a computer processor, a first lookup tablerepresenting a pixel-to-pixel mapping of source pixel coordinates of aninput image to destination pixel coordinates of an output image;generating, by the computer processor, a second lookup tablerepresenting a vector-to-pixel mapping of a first plurality of vectorsof the destination pixel coordinates to a first plurality of the sourcepixel coordinates based on the first lookup table, wherein all of thedestination pixel coordinates in each of the first plurality of vectorscorrespond to pixel-to-pixel mappings in the first lookup table that arelinear with slope 1 with respect to each other; generating, by thecomputer processor, a third lookup table representing a listing of asecond plurality of vectors of the destination pixel coordinates,wherein at least one of the destination pixel coordinates in each of thesecond plurality of vectors correspond to a pixel-to-pixel mapping inthe first lookup table that is not linear with slope 1 with respect toother pixel-to-pixel mappings; receiving, by the computer processor,source image data; applying, by the computer processor, a linearremapping operation from the source image data to each of thedestination pixel coordinates in the first plurality of vectors in thesecond lookup table to obtain a first portion of destination image data;and applying, by the computer processor, a non-linear remappingoperation from the source image data to each of the destination pixelcoordinates in the second plurality of vectors in the third lookup tableto obtain a second portion of the destination image data.
 2. The methodof claim 1, wherein the linear remapping operation comprises retrieving,by the computer processor and from the source image data, first sourcepixel data corresponding to a respective one of the first plurality ofvectors in the second lookup table based on the vector-to-pixel mappingin the second lookup table.
 3. The method of claim 2, further comprisingperforming, by the computer processor, an interpolation on the firstsource pixel data to obtain the first portion of the destination imagedata.
 4. The method of claim 1, wherein the non-linear transformationoperation comprises retrieving, by the computer processor and from thesource image data, second source pixel data corresponding to arespective one of the second plurality of vectors in the third lookuptable based on the pixel-to-pixel mapping in the first lookup table. 5.The method of claim 4, further comprising performing, by the computerprocessor, an interpolation on the second source pixel data to obtainthe second portion of the destination image data.
 6. The method of claim1, wherein all of the pixel-to-pixel mappings in the first lookup tablecomply with the following equation for all of the destination pixelcoordinates in each of the first plurality of vectors:F({x+i, y})=F({x, y})+{i,0}
 7. The method of claim 1, wherein each ofthe first and second pluralities of vectors comprises at least twoadjacent destination pixel coordinates.
 8. A non-transient computerprogram product having instructions encoded thereon that when executedby one or more processors cause a process to be carried out, the processcomprising: receiving a first lookup table representing a pixel-to-pixelmapping of source pixel coordinates of an input image to destinationpixel coordinates of an output image; generating a second lookup tablerepresenting a vector-to-pixel mapping of a first plurality of vectorsof the destination pixel coordinates to a first plurality of the sourcepixel coordinates based on the first lookup table, wherein all of thepixel-to-pixel mappings in the first lookup table are linear with slope1 for all of the destination pixel coordinates in each of the firstplurality of vectors; generating a third lookup table representing alisting of a second plurality of vectors of the destination pixelcoordinates, wherein any of the pixel-to-pixel mappings in the firstlookup table are not linear with slope 1 for any of the destinationpixel coordinates in each of the second plurality of vectors; receivingsource image data; applying a linear remapping operation from the sourceimage data to each of the destination pixel coordinates in the firstplurality of vectors in the second lookup table to obtain a firstportion of destination image data; and applying a non-linear remappingoperation from the source image data to each of the destination pixelcoordinates in the second plurality of vectors in the third lookup tableto obtain a second portion of the destination image data.
 9. Thecomputer program product of claim 8, wherein the linear remappingoperation comprises retrieving, from the source image data, first sourcepixel data corresponding to a respective one of the first plurality ofvectors in the second lookup table based on the vector-to-pixel mappingin the second lookup table.
 10. The computer program product of claim 9,wherein the process further comprises performing an interpolation on thefirst source pixel data to obtain the first portion of the destinationimage data.
 11. The computer program product of claim 8, wherein thenon-linear transformation operation comprises retrieving, from thesource image data, second source pixel data corresponding to arespective one of the second plurality of vectors in the third lookuptable based on the pixel-to-pixel mapping in the first lookup table. 12.The computer program product of claim 11, wherein the process furthercomprises performing, by the computer processor, an interpolation on thesecond source pixel data to obtain the second portion of the destinationimage data.
 13. The computer program product of claim 8, wherein all ofthe pixel-to-pixel mappings in the first lookup table comply with thefollowing equation for all of the destination pixel coordinates in eachof the first plurality of vectors:F({x+i, y})=F({x, y})+{i, 0}
 14. The computer program product of claim8, wherein each of the first and second pluralities of vectors comprisesat least two adjacent destination pixel coordinates.
 15. A systemcomprising: a storage; one or more computer processors operativelycoupled to the storage, the one or more computer processors configuredto execute instructions stored in the storage that when executed causeany of the one or more computer processors to carry out a processcomprising: receiving a first lookup table representing a pixel-to-pixelmapping of source pixel coordinates of an input image to destinationpixel coordinates of an output image; generating a second lookup tablerepresenting a vector-to-pixel mapping of a first plurality of vectorsof the destination pixel coordinates to a first plurality of the sourcepixel coordinates based on the first lookup table, wherein all of thepixel-to-pixel mappings in the first lookup table are linear with slope1 for all of the destination pixel coordinates in each of the firstplurality of vectors; generating a third lookup table representing alisting of a second plurality of vectors of the destination pixelcoordinates, wherein any of the pixel-to-pixel mappings in the firstlookup table are not linear with slope 1 for any of the destinationpixel coordinates in each of the second plurality of vectors; receivingsource image data; applying a linear remapping operation from the sourceimage data to each of the destination pixel coordinates in the firstplurality of vectors in the second lookup table to obtain a firstportion of destination image data; and applying a non-linear remappingoperation from the source image data to each of the destination pixelcoordinates in the second plurality of vectors in the third lookup tableto obtain a second portion of the destination image data.
 16. The systemof claim 15, wherein the linear remapping operation comprisesretrieving, from the source image data, first source pixel datacorresponding to a respective one of the first plurality of vectors inthe second lookup table based on the vector-to-pixel mapping in thesecond lookup table.
 17. The system of claim 16, wherein the processfurther comprises performing an interpolation on the first source pixeldata to obtain the first portion of the destination image data.
 18. Thesystem of claim 15, wherein the non-linear transformation operationcomprises retrieving, from the source image data, second source pixeldata corresponding to a respective one of the second plurality ofvectors in the third lookup table based on the pixel-to-pixel mapping inthe first lookup table.
 19. The system of claim 18, wherein the processfurther comprising performing, by the computer processor, aninterpolation on the second source pixel data to obtain the secondportion of the destination image data.
 20. The system of claim 15,wherein all of the pixel-to-pixel mappings in the first lookup tablecomply with the following equation for all of the destination pixelcoordinates in each of the first plurality of vectors:F({x+i, y})=F({x, y})+{i, 0}